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Aided Targeting System Simulation Evaluation 


JOE DE MAIO * AND CURTIS B ECKER+ 
Ames Research Center 


Summary 

Simulation research was conducted at the Crew Station 
Research and Development Facility on the effectiveness 
and ease of use of three targeting systems. A manual 
system required the aviator to scan a target array area 
with a simulated second generation forward looking 
infrared (FLIR) sensor, locate and categorize targets and 
construct a target hand-off list. The interface between the 
aviator and the system was like that of an advanced scout 
helicopter (manual mode). Two aided systems detected 
and categorized targets automatically. One system used 
only the FLIR sensor, and the second used FLIR fused 
with Longbow radar. The interface for both was like that 
of an advanced scout helicopter aided mode. Exposure 
time while performing the task was reduced substantially 
with the aided systems, with no loss of target hand-off list 
accuracy. The fused sensor system showed lower time to 
construct the target hand-off list and a slightly lower false 
alarm rate then the other systems. A number of issues 
regarding system sensitivity and criterion, and operator 
interface design are discussed. 

1. Introduction 

The purpose of the simulation was to demonstrate and 
evaluate the potential performance benefits that might be 
achieved through the use of an aided target processing 
capability. The immediate issue was the relative effect 
of system sensitivity and detection criterion on total 
accuracy, execution timeline, and operator acceptance. 
Two generic systems were investigated, a fused sensor 
system combining forward-looking infrared (FLIR) and 
radar sensors and a single sensor, FLIR system. 

We focused on two design parameters of aided targeting 
systems, sensitivity (d‘) and criterion (b). Sensitivity is of 
interest because it is a parameter which is substantially 
improved when two sensors operating in different spectral 
bands are fused by the aided targeting system. Criterion is 
of interest because the relative frequency of false alarms 
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can greatly affect the usability and user acceptance of a 
detection system. 

The sensitivity of a fused sensor system exceeds that of a 
single sensor system both because the radar provides 
more accurate range data and because the fused sensor 
system uses multiple data sources. A single sensor system 
operates using only coarse, target range data obtained 
either by lasing to the center of a FLIR frame or by 
triangulation. These relatively poor range data can lead to 
template matching ambiguities. Radar range data for 
individual target objects can eliminate many potentially 
erroneous template matches. Further ambiguity reduction 
comes because voting schemes can take advantage of 
correlations in the data from the two sensors. 

An important reason for increasing a system’s sensitivity 
is to reduce the rate or absolute number of false alarms it 
produces. False alarms by the aided system increase 
operator workload, and they also lower performance by 
impairing the operator’s vigilance. For any system the 
probability of detection must be traded off against the 
false alarm rate to provide the greatest possible number of 
detections without impairing operator performance. 

Another factor of interest in the test was the density of 
target objects in the environment. More target objects 
mean more detections. More detections increase operator 
tasking, and they may alter the way in which the system 
presents detections to the operator. Our concern was to 
determine simply whether target object density is a 
concern in evaluating aided targeting systems. 

While we understand that systems such as those evaluated 
here are intended for use on a wide variety of tasks per- 
formed under a broad range of conditions, we did not 
perform a parametric evaluation of aided targeting 
systems over their full range of application. Rather we 
selected a likely combination of task and environment 
which would be amenable to performing an evaluation in 
simulation. The characteristics of the test environment 
included high thermal contrast (high A T), giving a high 
quality FLIR image, and stationary targets. These 
conditions permitted good performance by both the 
aviator and the system and an adequate level of 
experimental control. 
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2. Method 

Experimental Design 

The simulation was designed to highlight the effects of 
automation on target detection and categorization per- 
formance. Two aided systems were studied. One was a 
multi-spectral system using passive forward-looking 
infrared (FLIR) and active radar. The other was a single 
sensor system using only FLIR. The actual systems would 
differ in a number of performance characteristics. These 
characteristics include detection sensitivity, ability to 
categorize targets, ability to provide accurate range data, 
susceptibility to environmental effects, and signature 
magnitude. The present study focused only on detection 
sensitivity and target categorization. 

Targeting System Modeling 

The end-to-end performance of a detector is determined 
by its sensitivity and by its signal detection criterion, and 
both factors must be considered in evaluating systems. 
The P(d) is the probability that a given target is detected 
on a given scan. For a specified system sensitivity, P(d) is 
determined only by the detection criterion. A high cri- 
terion leads to a low P(d). In effect desired P(d) defines 
the criterion for each system. When the criterion has been 
set to achieve a desired P(d), sensitivity determines FAR. 
A lower sensitivity system produces more false alarms for 
a given P(d) than does a higher sensitivity system. A 
notional illustration of the relationship between the 
sensitivity of the detector (d'), P(d) and FAR is presented 
in the receiver operating characteristic (ROC) shown in 
figure 2-1. The ROC relates the P(d) of a detector to its 


FAR. Since these two parameters vary with criterion, a 
sufficiently high criterion will result in no detections and 
no false alarms and a sufficiently low criterion will result 
in a P(d) of 1 .0 and a probability of false alarm given no 
target of 1.0. Between these extremes, detections and 
false alarms will vary to produce the monotonic function 
shown in figure 2-1. The degree of curvature of the ROC 
is proportional to d'. 

In the present study, two levels of correct detection 
performance were set. The probability of detection (P(d)) 
determined criterion based on system sensitivity, and a 
corresponding false alarm rate was determined. Thus 
system sensitivity and signal detection criterion were 
covaried to yield four representative combinations of 
correct detections and false alarms, specified on a per 
scan basis. With a scan field of regard (FOR) of 6° x 45° 
centered 2° below the horizon, the correct detection and 
false alarm rates (FAR) shown in table 2-1 obtain. 

“Target objects” were placed in the data base to be 
detected, with the number of target objects varied across 
vignettes (see table 2-4 in Operator Interface section). 
“Non-target objects” were placed in the data base to yield 
the false alarm rates per scan shown in table 2-1. FAR is a 
function of system sensitivity and detection criterion. 

The P(d) shown in table 2-1 is an average over all targets 
(target and non-target objects are treated identically by 
the targeting system). The P(d) for an individual object 
was determined by a function of P(d)j^ (see eq. 2-1), the 

Low 



Figure 2-1. Sample receiver operating characteristic. 


Table 2-1. Targeting system configurations 



High sensitivity 

Low sensitivity 

High 

P(d) = 0.6 

P(d) = 0.6 

criterion 

FAR = 0.6 

FAR = 3.6 

Low 

II 

o 

VO 

P(d) = 0.9 

criterion 

FAR = 4.5 

FAR = 19.8 
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average P(d), range and a detectability factor (three 
levels). Tanks had a detectability factor of 0.02. APCs 
and tracked ADUs had a detectability factor of 0.0. 
Wheeled vehicles had a detectability factor of -0.02. 
Similarly non-target objects had a detectability factor, 
0.02 for living (cow), 0.0 for man-made (dead vehicles), 
-0.02 for non-living (rocks and dead trees). The range 
scale factor was 0.08 at minimum range (about 1 km) and 
0.0 at 2 km. Individual probability of detection was the 
sum of the average P(d), detectability, and range scalar. 
The range of individual P(d) was 0.5 to 0.7 for the high 
criterion conditions and 0.8 to 1.0 for the low criterion 
conditions. Detectability factors associated with each 
type of target object and non-target object are shown in 
table 2-2. Objects were placed in the data base to give an 
average detectability factor across all objects of 0. This 
scheme allows variation in P(d) independent of average 
P(d). Eighty percent of the variation in P(d) within a 


scan results from range effects, while 20% results from 
variation in the targets. The P(d) of object j was given by 
the following formula: 

P(d)i=P(d) M + d + r (2-1) 

where 

d = {-0.02, 0.0, +0.02} 

r = -0.08 + {0.16 * [(range* - range m i n ) / (range max 
- range min )]} 

The sensor model was intended to provide a qualitatively 
correct rendering of the effects of target type and range. 
The target detection systems were modeled through a 
simple Monte Carlo process. Each target in the system 
FOR was detected through an independent stochastic 
process using P(d) determined by equation 2-1. Target 
objects and non-target objects were detected according to 


Table 2-2. Experimental targets 


Target type 

Classification type 

Recognition type 

Detectability factor 

Ml 

Tracked 

Tank 

0.02 

M60 

Tracked 

Tank 

0.02 

M3 

Tracked 

APC 

0.0 

M113 

Tracked 

APC 

0.0 

HUMMV 

Wheeled 

Truck 

-0.02 

T72 

Tracked 

Tank 

0.02 

BTR 

Wheeled 

Truck 

-0.02 

BMP 

Tracked 

APC 

0.0 

ZSU 

Tracked 

ADU 

0.0 

SA9 

Wheeled 

ADU 

-0.02 

Zil 

Wheeled 

Truck 

-0.02 

Rock 

- 

- 

-0.02 

Rock 

- 

- 

-0.02 

Rock 

- 

- 

-0.02 

Dead truck 

- 

- 

0.0 

Dead tree 

- 

- 

-0.02 

Dead hind 

- 

- 

0.0 

Dead tank 

- 

- 

0.0 

Cow 

- 

- 

0.02 
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the same stochastic process. Appropriate FARs were 
obtained by varying the number of non-target objects 
placed in the data base. In the experimental context the 
differences in individual P(d) provided a basis for target 
prioritization as well as the detection performance of the 
systems. If a target were detected, the individual P(d) 
would be used for target prioritization, with higher P(d) 
targets presented first. 

The aided targeting systems provided information about 
target type. The single sensor system indicated target 
classification. The fused sensor system provided recog- 
nition information. The target labels used are shown in 
table 2-3. The probability of correct categorization 
conditional on target detection was 0.6 for both systems. 


Table 2-3. Aided system categorization output 


Single sensor 
classification 

Multi-sensor 

recognition 

Tracked 

Tank 

Wheeled 

Truck 

Unknown 

ADU 


APC 


Unknown 


A manual targeting system was included as a baseline for 
comparison against the aided systems. The manual system 
used the same FLIR and laser ranging system as the 
single sensor aided system, but these subsystems were 
under manual control. The FLIR sensor had a selectable 
field of view (FOV) of 1.5° x 2° (narrow) or 6° x 8° 
(medium). The laser range finder was triggered manually 
to obtain a range for each target included in the target 
hand-off list. Since the manual system detected no targets 
and generated no false alarms, there could be no specified 
FAR, but we assumed that in the worst condition the 
aided system detected every possible non-target object in 
the environment. Therefore, the non-target object count 
for single sensor, low criterion condition was used for the 
manual condition. 

Target environment- Real targets were computer 
generated models of the military vehicles listed in 
table 2-2. Objects were placed into correlated out-the- 
window (OTW) visual and FLIR data bases. The OTW 
scene was created on a General Electric Compu-scene IV 
image generator. The FLIR imagery was generated on 
a Silicon Graphics Reality Engine. Vehicles were sta- 
tionary, and their placement conformed to tactics and 


doctrine as reflected in the Rotorcraft Pilot’s Associate 
vignettes. There were four conditions of target environ- 
ment, differing in the number of real targets placed in the 
environment. A target-free condition contained no target 
objects, and the only detections reported by the aided 
systems were non-target objects. A low target density 
condition corresponded to a combat reconnaissance 
platoon. This condition contained six targets. A medium 
density condition, corresponding to a regimental recon- 
naissance company, contained 10 targets. The high target 
density condition contained 14 targets. This condition 
corresponded to an advanced guard. 

The experimental design is shown in table 2-4. Two 
factors in the table taken together define the aided 
targeting system. The P(d) identically determines the hit 
rate, and it plus the number of non-target objects define 
the false alarm rate (number of non-target objects 
detected on a scan). A second environmental factor, 
number of target objects, defined the operational environ- 
ment of the test. Four target object densities were 
presented. 

Operator Interface 

Targeting display page- The targeting task was 
performed on a display page presented on the tactical 
situation display (TSD) of the CSRDF. The TSD consists 
of a 12 inch high-resolution, color CRT with an infrared 
touch screen. Subjects touched the area of the display 
showing the desired softkey control to activate the 
control. Touchscreen controls were of three types, 
instantaneous, on/off, and alternate action. In the 
following discussion controls are indicated by BOLD 
CAPITAL letters. 

The targeting display page consisted of a system control 
area along with imagery and controls specific to the 
targeting system. The system control area for targeting 
systems is shown in figure 2-2. This area would allow for 
selection of manual sensor scan and aided scan if this 
function were supported. Top level controls were 
continuously displayed around the side of the targeting 
display page. Imagery and lower level controls were 
displayed in the center. Labeling controls were displayed 
at the bottom. For the purposes of the present test, manual 
and aided scan modes were mutually exclusive. Therefore 
the controls for selecting one or the other were active only 
if that test condition were available. Since switching 
between modes was not an issue, highlighting of available 
modes was not provided. 

Manual system controls -The MANUAL FLIR softkey 
activated the sensor slewing control for manual mode 
targeting. Two sensor fields of view (FOV) were selec- 
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table by a toggling softkey, MEDIUM (6° x 8°) and 
NARROW (1.5° x 2°). The right side hand controller 
allowed the aviator to slew the sensor line of sight left 
and right and up and down over a range of ±1 10° in 
azimuth and +20°/-60° in elevation. An arrow in the 
lower left corner indicated the sensor line of sight in 
azimuth relative to the armament datum line (ADL). The 
aviator selected a target by touching the image on the 
sensor image display, and the system responded by 


placing a complete box around the target with the label 
“UNK” (unknown, fig. 2-3) and bringing up first level 
(classification) LABELING controls (fig. 2-4). The 
aviator could label the target at this level or select 
REC(ognition) or ID(entification). The aviator selected 
a label from the set shown in figure 2-4. Previously 
accepted targets that were not currently selected were 
indicated by a broken box. 


Table 2-4. Experimental conditions 


Sensor/ 
criterion a 

Target 

density 

p(d) 

Average 

targets 

detected 

(hits) 

Average 

number 

false 

alarms 

Number 

real 

targets 

Number 

false 

targets 

Number 

total 

targets 

F/H 

None 

0.6 

0 

0.6 

0 

1 

1 

F/H 

Low 

0.6 

3.6 

0.6 

6 

1 

7 

F/H 

Medium 

0.6 

6 

0.6 

10 

1 

11 

F/H 

High 

0.6 

8.4 

0.6 

14 

1 

15 

F/L 

None 

0.9 

0 

4.5 

0 

5 

5 

F/L 

Low 

0.9 

5.4 

4.5 

6 

5 

11 

F/L 

Medium 

0.9 

9 

4.5 

10 

5 

15 

F/L 

High 

0.9 

12.6 

4.5 

14 

5 

19 

S/H 

None 

0.6 

0 

3.6 

0 

6 

6 

S/H 

Low 

0.6 

3.6 

3.6 

6 

6 

12 

S/H 

Medium 

0.6 

6 

3.6 

10 

6 

16 

S/H 

High 

0.6 

8.4 

3.6 

14 

6 

20 

S/L 

None 

0.9 

0 

19.8 

0 

22 

22 

S/L 

Low 

0.9 

5.4 

19.8 

6 

22 

28 

S/L 

Medium 

0.9 

9 

19.8 

10 

22 

32 

S/L 

High 

0.9 

12.6 

19.8 

14 

22 

36 

MAN 

None 

n/a 

n/a 

n/a 

0 

22 

22 

MAN 

Low 

n/a 

n/a 

n/a 

6 

22 

28 

MAN 

Medium 

n/a 

n/a 

n/a 

10 

22 

32 

MAN 

High 

n/a 

n/a 

n/a 

14 

22 

36 


fl “F” and “S” refer to fused sensor and single sensor aided systems. “MAN” refers to the manual targeting 
system. “H” and “L” refer to high and low detection criteria. 
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Figure 2-3. Manual system targeting display page. 



Aided system control and display- The base control 
page contained softkeys for operating the aided targeting 
systems. INIT SCAN caused the system to execute a two 
pass scan (near and far bar). For the present test the area 
scanned was always 22.5° left and right of the ADL and 
1 ° above/5 ° below it, so no scan limit controls were 
provided. 

Following scan completion, the REVIEW softkey 
allowed the aviator to review the system detections. Four 
review mode controls were available on the right side of 
the page. CHIPS allowed the aviator to view up to 
18 detections, presented in groups of six (fig. 2-5). The 
P(d) of each was used to prioritize detections, with 
highest priority (highest P(d)) shown in chip number one. 
Detections with priority of 19 or below were not pre- 
sented in chips mode. Three frames modes allowed the 
aviator to view each of the 92 FLIR frames produced by 
the scans, with detections highlighted (fig. 2-6). AUTO 
FRAME first presented the frame in the upper left. At 
20 second intervals the system automatically stepped one 
frame, moving left to right and down. Any input by the 
aviator reset the interval counter. TGT FRAME per- 
formed similar stepping but only between frames contain- 
ing detections, beginning with the upper, left-most frame 
containing a detection. MAN FRAME did not step 
automatically, but the aviator could step manually using 
the UP, DOWN, LEFT, and RIGHT softkeys. These 
keys were also active in the automatic stepping modes. 

A grid displayed below the sensor imagery showed the 
frames array. The currently displayed frame was filled. 
Frames containing detections were stippled. 

Target selection and labeling- A target chip could be 
selected by touching the chip, after which YES or NO 
would accept or reject the detection. Alternatively YES 
or NO could be touched to select and accept or reject 
simultaneously. Label controls and a system-determined 
categorization became available when a detection was 
accepted. The initial level of categorization was classifi- 
cation for the single sensor system and recognition for the 
fused sensor system. 

In frames mode the bottom comers of a box indicated a 
system detection, a broken box indicated a detection that 
had been accepted, and a complete box indicated the 
selected target. A bullet (• ) indicated an object that the 
aviator had rejected as a target. A target object could be 
selected by touching it whether the system had detected it 
or not. If the system had detected it or it had previously 
been accepted, it could also be selected using the PREV 
TGT and NEXT TGT softkeys. Label controls were the 
same as for chips mode. 

Target hand-off list- Objects labeled by the aviator as 
target objects or non-target objects were automatically 


placed in a target hand-off list. This list was displayed on 
a 6 inch CRT located on the right side of the cockpit. For 
each object in the list the display presented a sequence 
number, the object label, and the object location (in 
Universal Transverse Mercator coordinates). Up to five 
target entries could be displayed at one time. PAGE UP 
and PAGE DN moved the display up and down the list in 
groups of five. “ A” and “V” moved up and down the list 
one object at a time. The SEND softkey terminated the 
experimental trial. 

Experimental Task 

The task presented to the experimental subjects was to 
reconnoiter a target area for which they have no prior 
intelligence and to formulate and transmit a target hand- 
off list. An experimental trial began at a point along the 
ingress route to the target area. The terminal phase on the 
ingress (about one minute) was prerecorded, and the 
experimental aircraft flew the subject to the designated 
observation point. The subject wore the fiber-optic helmet 
mounted display (FOHMD) to provide out-the-window 
cues for orientation to the task environment. At the end of 
the ingress, he called for an unmask, and the aircraft 
executed a prerecorded unmask to a hover. The aircraft 
was oriented with the target area directly in front of it. 

The subject executed his targeting task. In the aided 
conditions this activity consisted of initiating the scan. 

The subject then commanded a reipask when the scan was 
completed, after which he constructed and transmitted the 
target list. The process of constructing the list consisted of 
reviewing the output of the aided system, accepting or 
rejecting each detection and providing categorization to 
the highest level possible. The trial ended at message 
transmission. In the manual condition the subject slewed 
the FLIR sensor and entered target data and transmitted 
the target hand-off list. The trial ended with message 
transmission. 

Subjects 

Fourteen aviators participated in the experiment. Two test 
pilots were supplied by the Night Vision and Electronic 
Sensors Directorate (NVESD). These two aviators served 
from 07 SEP to 1 1 SEP 94. Their primary role was to 
support final debug of simulation and procedures, and 
their data were not analyzed. Twelve subjects were 
operational AH-64 aviators. These aviators had no 
CSRDF experience, but they were experienced in the 
AH-64 combat mission simulator. These aviators served 
in three groups of four each. Each group spent one week 
at the CSRDF during the period from 12 SEP through 
02 OCT 94. Experience of the 12 aviators is shown in 
table 2-5. 
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Figure 2-5. Chips mode display. 
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Figure 2-6. Frames mode display. 






























Table 2-5. Experimental aviator operational experience 


Aviator 

ID 

AH-64 

front seat time 
(hr) 

AH-64 
back seat 
time (hr) 

Attack 
helicopter 
time (hr) 

Total 
helicopter 
time (hr) 

1 

200 

150 

600 

750 

2 

160 

40 

500 

580 

3 

300 

70 

450 

700 

4 

300 

300 

2000 

3500 

5 

250 

400 

1500 

1700 

6 

200 

50 

300 

350 

7 

150 

20 

170 

350 

8 

150 

30 

300 

465 

9 

150 

50 

475 

550 

10 

50 

50 

1600 

2000 

11 

85 

40 

500 

700 

12 

170 

20 

190 

350 


Procedure 

Data collection took place over three days for each group 
of aviators. Each aviator performed two data collections 
per day (roughly two hours total), being exposed once to 
each system X target density condition. Prior to starting 
the data collection, the aviators received training in the 
operation of the simulated targeting systems, and they 
were fitted with a custom liner for the FOHMD. Aviators 
were trained in the operation of the targeting systems as 
well as in recognition of the CSRDF FLIR imagery of 
target ground vehicles. Aviators were trained until they 
and the test administrator agreed that they were proficient. 

Data collection took place on Tuesday, Wednesday and 
Thursday of each week. Two aviators were scheduled 
each morning session, and two were scheduled each 
afternoon. Each session consisted of two one-hour blocks 
per subject. Each subject completed two replications of 
each condition. 

Each one-hour block consisted of 10 data collection runs 
with the order of experimental conditions randomized. On 
each run the simulated aircraft was initialized a short 
distance from the observation point (OP). The targeting 
system was correctly configured, and the appropriate 
display page was presented on the back seat displays. The 
right hand display contained the target list display. The 


center display contained the sensor system display. The 
left hand display contained the map. The aviator also 
wore the FOHMD to aid in orienting to the tactical 
environment, but neither the OTW scene nor helmet 
displayed symbology was used in the targeting task. 

When the aviator signaled that he was ready, the pre- 
recorded run-in to the OP was executed. Data collection 
began when the aircraft achieved a masked position at the 
OP. The aviator signaled readiness to begin the targeting 
task. A prerecorded unmask to a hover was executed. 

The prerecorded flight sequence was frozen for as long 
as the aviator needed to complete his task, and then the 
sequence resumed with a remask. The aviator completed 
the targeting task and transmitted the target list. Data 
recording terminated with transmission of the target list, 
and the aviator provided a subjective evaluation of the 
targeting system (see appendix A). A final debriefing 
gave aviators an opportunity to comment on all aspects of 
the evaluation. 

Data Collection and Analysis 

Real time objective measures of performance were 
collected along with the subjective questionnaire data. 
Data on correct detections (hits), misses, false alarms, 
correct rejections and categorization accuracy came from 
the target hand-off list. An explicit action to reject each 
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non-target object detection (system false alarm) was 
required in order to support data collection. In addition 
objective measures of timing were determined. These 
included the following: 


Time to accept or reject each target 
Total time to target hand-off 
Exposure time 
Scanning sequence 


3. Results 

The objective of the simulation was to examine how 
system/configuration changes in an aided targeting 
system might affect the utility of that system. In order to 
address this issue, we have decomposed the issue into 
three component questions. These questions are (1) how 
does aiding affect the performance of the aviator on the 
targeting task, (2) how do changes in the performance 
of the aiding system affect aviator performance, and 
(3) what factors affect the interaction of the aviator with 
the system. These questions were addressed by way of 
analyses of variance on several dependent measures. A 
2x2 repeated measures ANOVA looked at aided system 
and criterion effects. A three-way repeated measures 
ANOVA looked at aided versus manual system. A 
2x2x4 repeated measures ANOVA looked at effects of 
target density on aided system performance. ANOVA 
tables are presented in appendix B. Significant effects are 
shown in bold for objective measures (B1-B13). Only 
significant effects are shown for subjective evaluations 
(B14-B18). 

The presentation of results is organized as follows. First 
we discuss the effect of aiding. Second we discuss how 
sensitivity of the aided systems affect the quality of 
performance and speed. Next we look at the effects of 
variation in criterion. We finish with a discussion of the 
effects of target density. A review of the aviators’ 


subjective evaluations supplements the analysis of the 
objective data. 


Effects of Aiding 

The major effect that could be attributed to aiding, 
per se, was a substantial reduction in exposure time 
(fig. 3-1). For this analysis exposure time was defined 
to be the period of time during which there was an 
unobstructed line of sight from the ownship to a vehicle 
in the target array being scanned. Exposure time with 
aiding was determined by the time required to enter a 
stable hover above the masking obstruction, to initiate 
the sensor scan manually, to complete the automated 
scanning process, and to descend below the obstruction. 
Exposure time in the manual condition was determined by 
the transition times as above plus the time required to 
scan, identify targets, and build the target hand-off list. 
Mean exposure time was 258 seconds for manual 
targeting. Exposure time was 37 to 61 seconds for the 
aided system, a reduction of over 75%. There was no 
difference between aided system configurations. 

Detection performance was measured by comparing the 
target hand-off list generated by the aviator with ground 
truth represented in the simulation data base. Detection 
performance was generally very good, since the FLIR 
simulation represented optimal viewing conditions. 
Overall the rate of correct detection was over 98%, that is, 
of 3095 target objects only 42 were missed. There were 
no differences in hit or miss rates for the various systems 
(fig. 3-2). The fused sensor system did show a slightly 
lower acceptance rate for false alarms than the single 
sensor and manual systems (fig. 3-3). Most of the false 
alarms were non-target vehicles that were most easily 
confused with target objects (fig. 3-4). The labeling of a 
non-target object as a target by the aiding system did not 
bias the aviator’s response to the non-target objects that 
were not visually confusable. 
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Figure 3-1. Exposure time for each targeting system. 
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Figure 3-2. Number of targets not included in target hand-off list. For manual condition 2 x N number is comparable to 
aided systems since manual sample is one-half as large. 
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Figure 3-3. Number of false alarms included in target hand-off list. 
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Figure 3-4. Number of non-target objects included in target hand-off list. 


14 




There was a difference in the reason for missed targets in 
the aided conditions as compared to the manual condition. 
In a number of cases, target objects were missed because 
they were not displayed to the aviator by the aided 
systems. More than one-half of the target objects missed 
with the aided systems were never presented to the 
aviator (fig. 3-5). Using the manual system, the aviator 
was presented every target object in the scene. 

Appendix C shows a typical range of scan fields for 
the aided and manual systems. 

The presentation of the manual scans in appendix C uses 
a denser pattern of dots to represent the narrow field of 
view (FOV) and a sparser pattern of dots to represent the 
medium FOV. When scanning manually, the aviators 
tended to concentrate on the main body of target objects 
or confusable non-target objects using the narrow FOV. 
They searched for the main body in the medium FOV, 
and they also used this FOV to search for outlying target 
objects after finishing the main body. This pattern of 
use is reflected in the small area of dense stippling sur- 
rounded by sparse stippling shown in the sample scans. 
Between narrow and medium FOV the sample scans 
cover an area between 150 square degrees and more than 
400 square degrees. The assigned area was 270 square 
degrees. 

The presentation of the aided scans indicates target 
objects by a “+,” target objects displayed in chips mode 
by a superimposed circle (“o”), and displayed frames by a 
stippled area. Uncircled target objects in an unfilled area 
were not presented to the aviator. All three examples of 
an aided scan contain undisplayed target objects. 

The sequence of operation with the aided systems was 
different than that used with the manual system. The 
aviators began by verifying all the detections presented in 


chips mode. If there were more than 18 detections, they 
verified the remaining detections in target frame mode. 
The aviators proceeded to subsequent targets by stepping 
sequentially left, right, up and down. They added any 
target objects missed by the system that they encountered 
during this search process. After completing the detec- 
tions, they made a search for outlying undetected target 
objects. This search was restricted to a small area around 
the main body of target objects, owing to the slowness 
and heavy manual involvement of stepping through 
frames. The sample scans cover an area between 
48 square degrees and slightly over 100 square degrees. 

Effects of Aided Systems 

Differences between and within aided systems showed 
their effects in the time needed to perform targeting 
operations. There were performance time differences 
resulting from differing sensitivity as well as differences 
arising from the type of operations performed by the 
aviator. 

On the whole the fused system supported much faster 
completion of the target hand-off list than did either the 
single sensor system or the manual system, which did not 
differ from each other. Time required to construct the 
target hand-off list (target hand-off time) was over 
50 seconds less with the fused system than with the other 
systems. This represents about a 20% reduction (fig. 3-6). 
For the aided systems, target hand-off time included time 
to process the chips array and the target objects presented 
in frames plus the time to search for outlying target 
objects plus the time to verify the target hand-off list. The 
process began as soon as the system presented a detected 
object to the aviator. Presentation of chips began during 
the scan, while the aircraft was still unmasked, but most 
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Figure 3-5. Proportion of missed target objects not presented to aviator. 
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of the time was spent after remask. By contrast the aviator 
performed the scanning in the manual condition, so the 
target hand-off time is equal to the exposure time. 

The differences in target hand-off time are interesting 
because they reflect not simply aiding but how well the 
aiding is done. The effectiveness of aiding depends on 
how the aviator must act on the data presented by the 
aiding system. The target hand-off time was shorter for 
the fused system even though the average time to process 
an individual detection was shorter for the single sensor 
system (fig. 3-7). This seeming paradox exists because 
the average time to process a detection masks important 
individual differences between detections. The time 
required to process each detection is determined by two 


factors, object type and presentation mode. Non-target 
objects are processed faster than target objects, because 
they require no labeling of target type, and chips 
presentations are processed faster than frames 
presentations (fig. 3-8). 

When the (single sensor) aided system produces more 
false alarms, the aviator must reject more non-target 
objects. Since this is the relatively faster process, the 
average time per object decreases. If, however, these 
objects were not detected, they would not be processed 
at all. Therefore the total time to accomplish the task 
increases by the amount required to process the added 
detections. 
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Figure 3-6. Time to construct and send target hand-off list. 
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Figure 3-7. Time to enter individual target data. 
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Figure 3-8. Time to enter individual target data by system mode and by target type. 


Increasing the number of false alarms also increases the 
total number of detections. When there are more than 
18 total detections, some are presented only in frames 
mode. Frames mode processing is slower than chips mode 
processing. Detected target objects displaced to frames 
mode presentation take longer to process, so the total time 
to perform the task increases. 

Chips mode processing is faster than frames mode 
processing because the former allows rapid selection of 
detections for processing while the latter requires a 
cumbersome search sequence. In chips mode up to six 
detections are presented as a group. The aviator can select 
by touching the imagery or the “yes” or “no” softkeys. 
The softkey is preferred because it required fewer actions. 
A single softkey touch brings up the next group of 
detections. In target frames mode the aviator finds 
detection number 19 and processes it. The aviator then 
steps one frame up or over. If there were a detection, he 
would select it and process it. If there were more he 
would select and process them; otherwise, he would step 
to another frame. This process is time consuming for 
several reasons. It requires a large amount of manual 
interaction with the system, it requires examination of a 
number of frames containing no detections, and it 
requires searching the frames to locate detections. 

An additional factor affecting processing time in frames 
mode is that missed target objects must be located in this 
mode. This task is essentially the same as the manual 
task, which is somewhat slower than processing detected 
objects. While this effect inflates our estimate of frames 
mode processing time slightly, it is still the case that the 
extra work associated with frames mode makes it 


substantially slower than chips mode for processing 
detections. 

Crew-System Interaction 

Crew-system interaction is generally considered to be an 
issue of physical interface design. Type and position of 
controls, display formats, operational sequences, and 
organization of data all affect usability and effectiveness. 
We did not vary the operator interface, but implemented 
that of the Comanche ATD/C. This design has not been 
fielded nor fully tested. We identified some difficulties in 
operating the aided systems using this interface. These are 
reported along with suggested improvements. 

A second crew-system interface issue for aided targeting 
systems is the criterion for target detection. For a given 
system the criterion determines the ratio of correct 
detections to false alarms. This ratio must be optimized 
for any given system sensitivity. The present study was 
not intended to optimize criterion. Rather it was intended 
to determine what aspects of performance are affected by 
criterion and how criterion interacts with other system 
design factors. 

Since criterion affects the ratio of correct detections to 
false alarms, we might expect it to affect performance 
time. In fact we see the same pattern of performance time 
effects for criterion that we saw for the difference in aided 
system sensitivity. That is, the lower criterion led to a 
longer time to perform the target hand-off task (fig. 3-9) 
even though the mean time to process an individual 
detection was shorter (fig. 3-10). This pattern occurs 
because the lower criterion generates more false alarms 
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(fig. 3-11). These are processed more rapidly than are 
correct detections, but because they increase the total 
number of detections to be processed, they lead to a 
longer total time to perform the task. The lower criterion 
was not unequivocally bad, however. Its benefit lay in 
reducing the number of targets that were missed because 
they were never presented to the aviator (fig. 3-12). As 
stated above target objects were not presented when the 
system failed to detect them and the aviator failed to 
select the appropriate frame during his stepwise search. In 
some sense the lower criterion compensates for the 
cumbersome frames mode interaction by obviating the 
need to find missed target objects. The cost of this 
compensation is a large increase in the time required to 
perform the target hand-off. This is a substantial cost, and 
the operator interface should be optimized to reduce the 
likelihood of failing to present target objects prior to 
optimizing criterion. Possible alternatives for optimizing 
the operator interface are discussed below. 

The effect of target density also turns out to be related to 
the aviator-system interaction. Increasing the number of 
target objects in the environment increases the target 
hand-off time, and this increase with target density is 


greatest for the lower criterion (fig. 3-13). The interaction 
between criterion and target density does not arise simply 
from there being more detections to process, since neither 
the single sensor nor the fused sensor system shows a 
sensitivity to target density relative to the manual system 
(fig. 3-14). Rather it is an effect of the operator interface 
design. As mentioned above, detections are processed 
more quickly in chips mode than they are in frames mode. 

Thus the time required to perform the hand-off increases 
when there are enough detections to result in some being 
presented only in frames mode. Table 3-1 shows the 
expected number of detections presented only in frames 
mode. All four target density conditions result in frames 
mode presentations for the single sensor, low criterion 
configuration. The highest target density condition results 
in frames mode presentation for the fused sensor, low 
criterion configuration and the single sensor, high 
criterion condition. All of these conditions show an 
elevated target hand-off time (fig. 3-15) except the no- 
target object, single sensor condition and the single 
sensor, high criterion condition. 



P(d) = 0.9 P(d) = 0.6 
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Figure 3-9. Time to target hand-off by criterion (P(d) = 0.9 is low criterion). 
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P(d) = 0.9 P(d) = 0.6 

PROBABILITY OF DETECTION 


Figure 3-10. Time to process individual detections by criterion (P(d) = 0.9 is low criterion). 
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Figure 3-11. Number of false alarms by criterion (P(d) = 0.9 is low criterion). 



PROBABILITY OF DETECTION 

Figure 3-12. Proportion of missed target objects not presented by criterion (P(d) = 0.9 is low criterion). 








Table 3-1. Number of detections presented in frames mode 


System 

configuration 

Number of 
target objects 

Number of non- 
target objects 

Expected 
number of 
detections 

Number 
presented in 
frames mode 

Fused, P(d) = 0.6 

0 

1 

0.6 

0 

Fused, P(d) = 0.6 

6 

1 

4.2 - 

0 

Fused, P(d) = 0.6 

10 

1 

6.6 

0 

Fused, P(d) = 0.6 

14 

1 

9.2 

0 

Fused, P(d) = 0.9 

0 

5 

4.5 

0 

Fused, P(d) = 0.9 

6 

5 

9.9 

0 

Fused, P(d) = 0.9 

10 

5 

13.4 

0 

Fused, P(d) = 0.9 

14 

5 

17.1 

0-1 

Single, P(d) = 0.6 

0 

6 

3.6 

0 

Single, P(d) = 0.6 

6 

6 

7.2 

0 

Single, P(d) = 0.6 

10 

6 

9.6 

0 

Single, P(d) = 0.6 

14 

6 

12.0 

0-2 

Single, P(d) = 0.9 

0 

22 

19.8 

0-4 

Single, P(d) = 0.9 

6 

22 

25.2 

0-10 

Single, P(d) = 0.9 

10 

22 

18.8 

0-14 

Single, P(d) = 0.9 

14 

22 

32.4 

0-18 



Figure 3-15. Target hand-off time: interaction between system configuration and target density. 



The frames mode effect is more apparent when the data 
are segregated by criterion than when they are segregated 
by system because all of the configurations in which 
detections are presented in frames are low criterion 
configurations while only four out of five of these are 
single sensor configurations. The effect is likely due to 
the frames mode operator interaction, not to the aided 
system. 

A further indication of the ineffectiveness of frames 
mode is the relatively large amount of time required to 


process target objects. Even though the aviators spent 
over twice as much time in frames mode as in chips mode 
(fig. 3-16), they processed fewer targets (fig. 3-17) in 
frames mode. It is true that area clearing and target list 
verification may inflate the time spent in frames mode by 
as much as 70 seconds, which is the mean time from the 
penultimate target acceptance to sending the list. Even 
accounting for this activity as much time, or more, is 
spent in frames mode to detect fewer targets. 
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Figure 3-16. Time spent in each system mode. 
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Figure 3- 1 7. Percentage of targets processed in each system mode. 
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Aviator Subjective Evaluation 

In general the aviators’ subjective evaluation of the 
systems was less sensitive to differences between the 
systems than was the objective data, but their evaluation 
paralleled the objective evaluation (table 3-2). They felt 
that the aided systems provided a substantial improve- 
ment in exposure time, although this improvement led to 
a perceived improvement in tactical value only for the 
fused sensor system. They felt that the fused system gave 
a shorter target hand-off time than either the single sensor 
system or the manual system and that the workload 
associated with the fused system was lower than with the 
manual system. They perceived the superior detection and 
categorization data provided by the aided systems, but, 
correctly, they did not feel that these resulted in better 
overall performance. 


Table 3-2. Aviator subjective evaluation^ 


Evaluation factor 

Fused 

sensor 

system 

Single 

sensor 

system 

Manual 

system 

Exposure time 

6.6 

6.6 

2.5 

System detection 

5.4 

4.6 

2.0 

System 

categorization 

3.8 

3.2 

1.8 

Target hand-off 
time 

5.3 

4.6 

4.1 

Workload 

5.7 

5.1 

4.8 

Tactical value 

5.8 

5.3 

4.9 


^Shows mean ratings. Seven represents the best rating 
and one represents the worst. 


4. Discussion 

Effects of Aiding 

The targeting system evaluation has demonstrated a clear 
advantage of aided systems over a manual system having 
the same FLIR sensing capability. The greatest benefit of 
the aided systems was the reduction in exposure time. 

This benefit had been anticipated, and in fact the exposure 
time for the aided systems was largely defined by the 
scan time of the system. The relative unknown was the 
exposure time for the manual system. At an average of 
over six minutes, the exposure time obtained is longer 
than would be considered tactically sound under actual 


combat conditions, but it is an accurate measure of the 
time needed to perform the complete targeting task. 
Tactical considerations might force segmentation of the 
task to reduce exposure time. Such segmentation could 
greatly increase the duration of a manual targeting task. 
Therefore the benefit derived from aiding might be sub- 
stantially greater under combat conditions than under the 
conditions in the present test. 

A significant concern about aided systems is that the 
improvement in exposure time might be purchased at the 
expense of poorer performance. This was definitely not 
the case. Overall the aided systems gave performance 
comparable to the manual system in accuracy with less 
risk due to exposure to ground threats. The fused sensor 
system actually allowed quicker construction of the target 
hand-off list, by about 50 seconds, than either the single 
sensor system or the manual system. The design of the 
test mitigated against finding accuracy improvements 
with the aided systems, since the high thermal contrast 
(high A T) image made the aviator’s visual discrimination 
task easy. Aiding could enhance accuracy under poor 
thermal contrast (low A T) conditions. We might expect 
the greatest benefit from the fused sensor system, since 
the Longbow would provide it information to compensate 
for the poor FLIR data. 

Crewstation Interface 

Chips mode - The chips mode interface was highly 
effective in allowing the aviators to process detections 
quickly. A large number of individual detections could be 
processed in this mode. It was also useful for verifying 
the target hand-off list. Its limitation was that only 
1 8 detections could be presented in this mode. This 
limitation was caused by processing and memory in 
available, flight qualified hardware, that should not limit 
future systems. 

Chips mode should be expanded to cover all system 
detections. Chips-like review modes would also be useful 
for verification of the target hand-off list. One such mode 
would include all accepted targets, whether detected by 
the system or by the crew. This mode would enable the 
aviator to verify the accuracy of the targets he has 
included in the target hand-off list. A second mode would 
present only detections not accepted by the crew as 
targets. This mode would facilitate the aviator’s checking 
his own editing. 

Chips mode presents only individual targets detected by 
the system. A substantial part of the reconnaissance task 
requires frames mode. Frames mode supports detection of 
target objects missed by the system. It also allows the 
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aviator to view global relationships between target 
objects. 

Frames mode- The aviators were no more likely to miss 
target objects with the aided systems than with the 
manual system, and in fact the miss rate was very low 
across the board. There was, however, a negative effect in 
that a number of misses with the aided systems resulted 
from the target object never having been displayed to the 
aviator. This situation occurred when the system failed to 
detect the object and the aviator failed to select the frame 
containing the object during his search for system misses. 
This problem was symptomatic of a design flaw in the 
operator interface, which has not yet been fielded nor 
fully tested. This flaw is the cumbersome process of 
sorting through a large number of sensor frames (92 for 
the 6° x 45° scan). The aviators found this process time 
consuming and monotonous. In addition, the rate of non- 
displayed system misses indicates that it is somewhat 
error prone. 

Further potential problems exist with the frames mode 
interface that were not directly apparent in the present 
evaluation. One is the possibility of missing outlying 
target objects due to failure of the aviator to view the 
entire search zone. Appendix C shows that the aviators 
reduced the area they actually viewed with the aided 
systems. Another potential problem is that relationships 
between targets in the scanned formation may be less 
apparent when the area is viewed in small frames. The 
present evaluation did not test this sort of situation 
awareness, but the aviators commented that they felt the 
fragmented view to be less than satisfactory. 

Aviators used the medium field of view in the manual 
mode to provide a big picture view. Having found 
possible target objects in the medium FOV, they switched 
to narrow FOV to gain adequate resolution for their 
decision. Frames mode should provide a comparable 
capability. Pixel expanded medium and wide FOV modes 
would allow the crew to search quickly and thoroughly, 
getting a global perspective. Narrow frames would 
provide a detailed image, and the chips-like modes would 
support review of the final product of the reconnaissance 
activity. 


act on the information provided to him by the system. 

As criterion varies, the aviator has to deal with varying 
numbers of hits and false alarms. We have seen that the 
system’s judgment does little to bias the aviator, at least 
under the high thermal contrast conditions of the present 
evaluation. The principal effect of criterion variation is to 
change the amount of frames mode interaction. Frames 
mode can lead to failure to present target objects when 
criterion is high. When criterion is low, frames mode 
increases the time needed to perform the reconnaissance. 
Both of these effects stem from the design problems cited 
above. A definitive judgment regarding criterion values 
must await optimization of system modes. 

Another interaction factor linked to the criterion issue is 
the crew’s response to detected non-target objects. We 
required an explicit response for data collection. This 
response slowed processing of detected non-target 
objects. A “no-target” default would speed the task and 
would likely have no adverse effect on accuracy. This 
change should be included in an operator interface 
refinement and evaluation program. 

5. Conclusions 

1. Aided targeting systems provide a substantial reduction 
(over 81%) in exposure to ground based threats. 

2. The fused sensor system provides a considerable 
reduction in time to construct the target hand-off list. 

3. The benefits of aiding are obtained with no loss of 
accuracy. The fused sensor system actually showed 
slightly lower false alarm rates than the other systems. 

4. Aiding may improve accuracy under poor FLIR 
viewing conditions (low A T), particularly with the fused 
sensor system. 

5. Battlefield conditions, which lead to greater time stress, 
may magnify the benefits obtained under the present, 
relatively benign test conditions. 

6. Operator interface/system mode refinement and 
evaluation should be an integral part of system design to 
address problem areas identified in the present test. 


Crew-System Interaction 

We have addressed crew-system interaction as a criterion 
issue. Detection criterion affects how the aviator has to 
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6. Appendices 




A. Aviator Subjective Evaluation Form 


PS#C*DtN« PAGE BLANK NOT FILMED 




Targeting System Evaluation Questionnaire 



Name: 




Run number: 

1. Rate the ease of use of each targeting svstem. 

Manual 1 2 

3 

4 

5 

6 

7 

very hard 

Single- 1 2 

3 

4 

5 

6 

very easy 
7 

Sensor very hard 

Multi- 1 2 

3 

4 

5 

6 

very easy 
7 

Sensor very hard 





very easy 

1 2. Rate the accuracv of each targeting svstem. 1 

Manual 1 2 

3 

4 

5 

6 

7 

very poor 

Single- 1 2 

3 

4 

5 

6 

very good 
7 

Sensor very poor 

Multi- 1 2 

3 

4 

5 

6 

very good 
7 

Sensor very poor 





very good 

3. Rate vour accuracv using each targeting svstem. 

Manual 1 2 

3 

4 

5 

6 

7 

very poor 

Single- 1 2 

3 

4 

5 

6 

very good 
7 

Sensor very poor 

Multi- 1 2 

3 

4 

5 

6 

very good 
7 

Sensor very poor 





very good 

4. Rate exposure time with each targeting svstem. 

Manual 1 2 

3 

4 

5 

6 

7 

very long 

Single- 1 2 

3 

4 

5 

6 

very short 
7 

Sensor very long 

Multi- 1 2 

3 

4 

5 

6 

very short 
7 

Sensor very long 





very short 

5. Rate total time to nerform targeting task. 

Manual 1 2 

3 

4 

5 

6 

7 

very long 

Single- 1 2 

3 

4 

5 

6 

very short 
7 

Sensor very long 

Multi- 1 2 

3 

4 

5 

6 

very short 
7 

Sensor very long 





very short 

6. Rate the tactical value of each svstem. 

Manual 1 2 

3 

4 

5 

6 

7 

very low 

Single- 1 2 

3 

4 

5 

6 

very high 
7 

Sensor very low 

Multi- 1 2 

3 

4 

5 

6 

very high 
7 

Sensor very low 





verv high 

f 
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B. Analysis of Variance Tables 


fmm. buwk mrr filled 




Number of observations in data set = 422 
Dependent Variable: EXPSD Total Time Aircraft was Exposed (seconds) 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

Model 

14 

3227380.10541434 

230527.15038674 

88.87 

0.0001 

Error 

407 

1055726.26571824 

2593.92202879 



Corrected Total 

421 

4283106.37113258 




R-Square 


C.V. 

Root MSE 

EXPSD Mean 

0.753514 


56.66843 

50.93056085 

89.87465599 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr>F 

PILOT 

10 

83536.94273476 

8353.69427348 

3.22 

0.0005 

TGTJM 

4 

3133994.89815930 

783498.72453983 

302.05 

0.0001 


Bl. ANOVA — Exposure Time. 


Number of observations in data set = 422 
Dependent Variable: N_R_N_D Number Real Targets NOT Detected 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

121 

13.66991075 

0.11297447 

0.66 

0.9936 

Error 

190 

32.67624310 

0.17198023 



Corrected Total 

311 

46.34615385 




R-Square 


C.V. 

Root MSE 

N_R_N_D Mean 

0.294952 


308.0666 

0.41470499 

0.13461538 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

1.28089976 

0.12808998 

0.74 

0.6816 

RUN 

39 

4.01890639 

0.10304888 

0.60 

0.9703 

TGT_M_J3 

2 

0.11084093 

0.05542047 

0.32 

0.7249 

RUN*TGT_M_J3 

70 

7.50981624 

0.10728309 

0.62 

0.9881 


B2. ANOVA — Misses by Targeting System. 
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Number of observations in data set = 422 
NOTE: Due to missing values, only 421 observations can be used in this analysis. 
Dependent Variable: N_F_ACC Number False Targets Accepted as Real 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

126 

692.54453316 

5.49638518 

4.16 

0.0001 

Error 

294 

388.37233145 

1.32099432 



Corrected Total 

420 

1080.91686461 




R-Square 


C.V. 

Root MSE 

N_F_ACC Mean 

0.640701 


140.6611 

1.14934517 

0.81710214 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

220.81298601 

22.08129860 

16.72 

0.0001 

RUN 

39 

105.03549591 

2.69321784 

2.04 

0.0005 

TGT_M_J3 

2 

90.94249257 

45.47124628 

34.42 

0.0001 

RUN*TGT_M_J3 

75 

147.18956166 

1.96252749 

1.49 

0.0113 


B3. AN OVA — False Alarms. 


Number of observations in data set = 422 
Dependent Variable: PROP_OF_MISSED_TARGETS_NOT_PRESENTED 


Source 

DF 

Sum of Squares 

Mean Square 

F Value Pr > F 

Model 

12 

0.33071382 

0.02755949 

0.85 0.5949 

Error 

409 

13.20483120 

0.03228565 


Corrected Total 

421 

13.53554502 



R-Square 


C.V. 

Root MSE 

PROP_M Mean 

0.024433 


541.6131 

0.17968208 

0.03:11753 

Source 

DF 

Type IE SS 

Mean Square 

F Value Pr > F 

PILOT 

10 

0.09808461 

0.00980846 

0.30 0.9801 

TGT_M_J3 

2 

0.22915471 

0.11457735 

3.55 0.0296 


B4. ANOVA — Number of Missed Target Objects not Presented. 



Number of observations in data set = 422 
Dependent Variable: TTH Time To Target Hand-off (seconds) 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

Model 

12 

1242489.51191388 

103540.79265949 

11.16 

0.0001 

Error 

409 

3793218.76751411 

9274.37351470 



Corrected Total 

421 

5035708.27942799 




R-Square 


C.V. 

Root MSE 

TTH Mean 

0.246736 


39.92129 

96.30354882 

241.23358899 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

864300.21394836 

86430.02139484 

9.32 

0.0001 

TGT_M_J3 

2 

363255.23426774 

181627.61713387 

19.58 

0.0001 


B5. ANOVA — Time to Target Hand-off. 


Number of observations in data set = 422 
Dependent Variable: TIME Time Required For Target Determination 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

12 

24760.95770457 

2063.41314205 

51.17 

0.0001 

Error 

5614 

226377.83178860 

40.32380331 



Corrected Total 

5626 

251138.78949317 




R-Square 


C.V. 

Root MSE 

TIME Mean 

0.098595 


80.28132 

6.35010262 

7.90981300 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

17898.76238181 

1789.87623818 

44.39 

0.0001 

TGT_M_J3 

2 

6187.06434307 

3093.53217154 

76.72 

0.0001 


B6. ANOVA — Time to Process an Individual Target. 
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Number of observations in data set = 422 
Dependent Variable: TIME Time Required For Target Determination 


Source 

Model 

Error 

Corrected Total 

R-Square 

0.259181 

Source 

PILOT 

REAL 

MODE 

MODE*REAL 


Source 

Model 

Error 

Corrected Total 

R-Square 

0.345505 

Source 

PILOT 

ATM_C 

ATM_S 

ATM_C*ATM_S 


DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

15 

65090.45764067 

4339.36384271 

130.87 

0.0001 

5611 

186048.33185249 

33.15778504 



5626 

251138.78949317 





C.V. 

Root MSE 

TIME Mean 


72.79919 

5.75827969 

7.90981300 

DF 

Type III SS 

Mean Square 

F Value 

Pr>F 

10 

13804.11564181 

1380.41156418 

41.63 

0.0001 

1 

8193.69244699 

8193.69244699 

247.11 

0.0001 

2 

9630.56492595 

4815.28246297 

145.22 

0.0001 

2 

3517.53880691 

1758.76940345 

53.04 

0.0001 


B7. ANOVA — Time to Enter Individual Target Data. 


Number of observations in data set = 422 
Dependent Variable: TTH Time To Target Hand-off (seconds) 


DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

13 

1399191.52145124 

107630.11703471 

13.03 

0.0001 

321 

2650511.78737057 

8257.04606657 



334 

4049703.30882181 





C.V. 

Root MSE 

TTH Mean 


38.38302 

90.86828966 

236.74083826 

DF 

Type in SS 

Mean Square 

F Value 

Pr>F 

10 

618816.03621018 

61881.60362102 

7.49 

0.0001 

1 

324094.27126135 

324094.27126135 

39.25 

0.0001 

1 

333486.26219446 

333486.26219446 

4039 

0.0001 

1 

89049.35866618 

89049.35866618 

10.78 

0.0011 


B8. ANOVA — Time to Target Hand-off by Criterion. 


36 



Number of observations in data set = 422 
Dependent Variable: TIME Time Required For Target Determination 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

14 

53336.69941862 

3809.76424419 

115.34 

0.0001 

Error 

4819 

159179.00186784 

33.03154220 



Corrected Total 

4833 

212515.70128646 




R-Square 


C.V. 

Root MSE 

TIME Mean 

0.250978 


76.17454 

5.74730739 

7.54491857 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr>F 

PILOT 

10 

18655.61402806 

1865.56140281 

56.48 

0.0001 

TM_C 

1 

18.40346889 

18.40346889 

0.56 

0.4554 

ATM_S 

1 

48.27414799 

48.27414799 

1.46 

0.2268 

ATM_C*ATM_S 

1 

103.59856558 

103.59856558 

3.14 

0.0766 

REAL 

1 

29092.16932434 

29092.16932434 

880.74 

0.0001 


B9. ANOVA — Time to Process Detections by System and Criterion. 


Number of observations in data set = 422 
Dependent Variable: N_F_ACC Number False Targets Accepted as Real 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

52 

375.81741717 

7.22725802 

6.51 

0.0001 

Error 

281 

311.83827146 

1.10974474 



Corrected Total 

333 

687.65568862 




R-Square 


C.V. 

Root MSE 

N_F_ACC Mean 

0.546520 


149.7236 

1.05344423 

0.70359281 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

163.22601576 

16.32260158 

14.71 

0.0001 

RUN 

39 

70.73831310 

1.81380290 

1.63 

0.0133 

ATM_C 

1 

30.80365069 

30.80365069 

27.76 

0.0001 

ATM_S 

1 

79.53399499 

79.53399499 

71.67 

0.0001 

ATM_C*ATM_S 

1 

8.91574605 

8.91574605 

8.03 

0.0049 


BIO. ANOVA — False Alarms by System and Criterion. 
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Number of observations in data set = 422 
Dependent Variable: TTH Time To Target Hand-off (seconds) 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

21 

2776553.81415053 

132216.84829288 

23.41 

0.0001 

Error 

400 

2259154.46527746 

5647.88616319 



Corrected Total 

421 

5035708.27942799 




R-Square 


C.V. 

Root MSE 

TTH Mean 

0.551373 


31.15338 

75.15241954 

241.23358899 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

897691.12591794 

89769.11259179 

15.89 

0.0001 

TGTJDENS 

3 

1306729.71218274 

435576.57072758 

77.12 

0.0001 

TGT_M_J3 

2 

339009.91596192 

169504.95798096 

30.01 

0.0001 

TGT_M_J3 

6 

52553.68400731 

8758.94733455 

1.55 

0.1603 


*TGT_DENS 

Bll. ANOVA — Effect of Target Density. 


Number of observations in data set = 422 
Dependent Variable: TTH Time To Target Hand-off (seconds) 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr>F 

Model 

25 

2726648.50579628 

109065.94023185 

25.47 

0.0001 

Error 

309 

1323054.80302553 

4281.73075413 



Corrected Total 

334 

4049703.30882181 




R-Square 


C.V. 

Root MSE 

TTH Mean 

0.673296 


27.63990 

65.43493527 

236.74083826 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 

PILOT 

10 

646497.28869393 

64649.72886939 

15.10 

0.0001 

TGT_DENS 

3 

1248183.65886381 

416061.21962127 

97.17 

0.0001 

TGT_M 

3 

741285.86665516 

247095.28888505 

57.71 

0.0001 

TGT_M*TGT_DENS 

9 

72619.05200043 

8068.78355560 

1.88 

0.0537 


B12. ANOVA — Effect of Target Density by Aided System Configuration. 



Number of observations in data set = 422 
Dependent Variable: TIMEM Time Spent In Mode (seconds) 


Source 

DF 

Sum of Squares 

Mean Square 

F Value 

Pr > F 

Model 

34 

10170726.80218550 

299139.02359369 

169.65 

0.0001 

Error 

2060 

3632420.40672497 

763.31087705 



Corrected Total 

2094 

13803147.20891050 




R-Square 


C.V. 

Root MSE 

TIMEM Mean 

0.736841 


96.04019 

41.99179535 

43.72314849 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr>F 

PILOT 

10 

153765.38166671 

15376.53816667 

8.72 

0.0001 

MODE 

4 

3489486.42048208 

872371.60512052 

494.73 

0.0001 

TGT_M 

4 

158064.56859834 

39516.14214958 

22.41 

0.0001 

MODE*TGT_M 

16 

6377055.47521187 

398565.96720074 

226.03 

0.0001 


B13. AN OVA — Time Spent in Mode. 


Source 

• SS 

df 

MS 

F 

P 

Between Blocks 

9.0626 

11 




Within Blocks 

147.8172 

24 




SENSOR 

130.0482 

2 

65.0241 

80.507 

0.0000 

Error 

17.7690 

22 

0.8077 



Total 

156.8798 

35 





B14. ANOVA — Subjective Evaluation of Exposure Time. 




Source 

SS 

df 

MS 

F 

P 

Between Blocks 

24.0024 

11 




Within Blocks 

117.2898 

24 




SENSOR 

77.4934 

2 

38.7467 

21.420 

0.0000 

Error 

17.7690 

22 

0.8077 



Total 

141.2922 

35 





B15. AN OVA — Subjective Evaluation of System Detection. 
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Source 

SS 

df 

MS 

F 

P 

Between Blocks 

18.4040 

11 




Within Blocks 

57.4687 

24 




SENSOR 

23.6204 

2 

11.8102 

7.676 

0.0033 

Error 

33.8483 

22 

1.5386 



Total 

75.8727 

35 





B16. ANOVA — Subjective Evaluation of System Categorization. 



Source 

SS 

df 

MS 

F 

P 

Between Blocks 

20.4431 

11 




Within Blocks 

14.2856 

24 




SENSOR 

4.2539 

2 

2.1264 

4.663 

0.0202 

Error 

10.0327 

22 

0.4560 



Total 

34.7286 

35 





B17. ANOVA — Subjective Evaluation of Workload. 




Source 

SS 

df 

MS 

F 

P 

Between Blocks 

18.3413 

11 




Within Blocks 

15.8502 

24 




SENSOR 

5.0052 

2 

2.5026 

5.077 

0.0152 

Error 

10.8450 

22 

0.4930 



Total 

34.1915 

35 





B18. ANOVA— Subjective Evaluation of Tactical Value. 


40 





C. Sample Sensor Scans 
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